Rasch in R and Quarto, part 1 (updated)

Intro and WAAQ-7

Magnus Johansson

2022-06-09

Why Rasch analysis in R?

  • Traceability
    • Transparency in decisionmaking
  • Reproducibility
  • Freely available
  • Documentation in the same file as analysis
    • (with Quarto/Rmarkdown)

Agenda overview

  • Sample analysis with provided data and code
    • WAAQ-7
    • PSS-10
  • Suggested structure for analysis
    • Confirmatory and/or exploratory

Quarto formatting basics

https://quarto.org/docs/authoring/markdown-basics.html

  • Keep the page above open in a web browser
  • We will use library(kableExtra) for tables
    • since it works with HTML/revealjs and PDF output
    • even though library(formattable) looks nicer…

Exercise 1

  • I’ll walk you through the Rstudio/Quarto interface
  • You’ll run the setup code chunk, and practice navigation a bit
    • Enable/disable Outline
    • Navigate with Outline
    • Navigate with chunks
    • Switch between source and visual editor
    • Changing size of the quadrant windows
    • Look at the objects df and itemlabels

Exercise 2

  • We will walk through the code chunks together
  • Learn what they do, but not how (in detail)
  • I’ll explain some basic R concepts as we move ahead
  • Focus on how you can structure the analysis
  • Discuss the output briefly
    • feedback on how to improve is welcome :)
  • Small exercises are described at the end of the file

Exercise 3 (later on)

  • In the folder “Exercise2” you will find a .qmd file
  • It contains a setup block that will import a dataset (PSS-14)
  • You can practice using this file (RaschR1) as a template
    • copy code chunks for your analysis from this file to the other file
    • copy code, run the chunks, check the output, write commentary. Repeat
    • there is a file called “Exercise2solution” that shows how I did the analysis

Background information

The Work-related Acceptance and Action Questionnaire (Bond, Lloyd, and Guenole 2013) is a measure of psychological flexibility contextualized to a workplace setting.

The WAAQ uses 7 questions, all using the same response scale with 7 steps (ranging from “Stämmer aldrig” to “Stämmer alltid”, with labels on all categories).

itemnr item
q1 Jag kan arbeta effektiv trots personliga problem som oroar mig
q2 Jag kan erkänna mina misstag i arbetet och ändå vara framgångsrik
q3 Jag kan arbeta mycket effektivt även när jag är orolig för någonting.
q4 Oro hindrar mig inte från att nå framgångar.
q5 Jag kan göra vad som krävs, oavsett hur jag mår.
q6 Jag kan arbeta effektivt även när jag tvivlar på mig själv.
q7 Mina tankar och känslor hindrar mig inte i mitt arbete.

Participants

We have 229 participants and their gender distribution is shown in the table below. All participants are Swedish adults and the dataset was collected to validate a measure of social interactions at workplaces (Johansson 2021; Johansson and Biglan 2021).

Gender distribution in sample
Gender n Percent
Female 76 33.2
Male 153 66.8

Descriptives of raw data

Responses to all items are summarized below.

Response category Number of responses Percent
0 17 1.1
1 87 5.4
2 150 9.4
3 354 22.1
4 521 32.5
5 386 24.1
6 88 5.5

Descriptives - item level

Recoding responses

There are fewer than 10 responses in category 0 for all items, which is problematic for the analysis. We will merge the two lowest categories. For item 2 the four lowest categories will be merged.

Some cells still have fewer than 10 responses. Let’s look at the response categories using Rasch modeling to produce ICC plots.

Analysis of response categories

Items 1 and 5 have issues with disordered response categories, which need to be addressed. For both items the two lowest categories will be merged.

ICC plots for recoded items

Response categories are working as expected, after the adjustments made.

Floor/ceiling effects - raw data

Dimensionality

The eRm package, which uses Conditional Maximum Likelihood (CML) estimation, will be used primarily. For this analysis, the Partial Credit Model will be used.

First, we look at the eigenvalues from PCA analysis of Rasch residuals. These should be below 2.0 to support unidimensionality. If the first eigenvalue is above 2.0, the subsequent analysis will include a clustering analysis, using Mokken Scaling.

PCA of Rasch model residuals
Eigenvalues
1.50
1.38
1.18
1.10
1.02

Since the eigenvalues support unidimensionality, we move on to look at item fit to the Rasch PCM.

Item fit

OutfitMSQ InfitMSQ OutfitZSTD InfitZSTD
q1 0.77 0.793 -2.608 -2.378
q2 1.369 1.297 3.691 3.125
q3 0.668 0.654 -3.854 -4.053
q4 0.964 0.913 -0.353 -0.92
q5 0.868 0.831 -1.356 -1.959
q6 0.851 0.776 -1.628 -2.609
q7 0.812 0.801 -2.058 -2.234

Values in red are below/above the pre-set cutoff values (. We’ll leave them for now.

Residual correlations

q1 q2 q3 q4 q5 q6 q7
q1
q2 -0.06
q3 0.09 -0.24
q4 -0.15 -0.06 -0.16
q5 -0.13 -0.25 -0.16 -0.16
q6 -0.21 -0.18 -0.13 -0.25 -0.12
q7 -0.26 -0.28 -0.09 -0.24 -0.07 -0.07
Note:
Relative cut-off value (highlighted in red) is 0.049, which is 0.2 above the average correlation.

A correlation matrix is created based on the Rasch model residuals for each item. The average correlation is calculated, and item pairs that correlate more than the pre-set cutoff value of 0.2 above the average correlation are indicated in red in the table below.

Items 1 and 3 are correlated above the relative cutoff value. Looking at the item descriptions, both are related to working efficiently when being worried in general (item 3) or being worried about personal problems. As we saw in the section on item fit, item 3 had quite low item fit values. This is a common issue with broad questions.

It would be reasonable to eliminate either question 1 or 3, since they clearly overlap in content. We will leave this for an exercise.

DIF analysis

Our sample is underpowered to conduct a proper DIF-analysis of any of the demographic variables available (gender/age). N > 250 per subgroup would be ideal, and N > 150 minimal.

For the sake of practice, we will go through the steps in setting up and conducting a DIF analysis anyway.

We will use the Rasch Tree tool (Strobl, Kopf, and Zeileis 2013), which makes it possible to evaluate interaction DIF, if sample size allows. For this analysis, we will only investigate gender effects on DIF.

[1] "No significant DIF found."

No DIF was found in our small sample.

Targeting & item/person locations

Three figures with the same scale on top of each other. At the top is a histogram of Person Locations, with a dotted line and gray field indicating mean/SD. In the middle is a similar histogram with Item Thresholds. At the bottom is a figure showing the individual item thresholds.

Three figures with the same scale on top of each other. At the top is a histogram of Person Locations, with a dotted line and gray field indicating mean/SD. In the middle is a similar histogram with Item Thresholds. At the bottom is a figure showing the individual item thresholds.

Reliability

“Test information” shows the collective information based on the combination of all items into a unidimensional scale.

The horizontal lines at TIF 3.33 and 5.0 indicate the conventional lower cutoff for acceptable reliability of 0.7 and 0.8, respectively.

The table footnote includes information on floor/ceiling effects.

Item information

“Item information” shows individual curves indicating the amount of information contributed by each item.

Person location and infit ZSTD

Item parameters

Threshold 1 Threshold 2 Threshold 3 Threshold 4 Threshold 5 Item location
q1 -1.60 -0.57 1.38 4.86 NA 1.02
q2 -1.21 0.67 3.85 NA NA 1.1
q3 -2.04 -1.67 0.20 2.19 5.80 0.9
q4 -1.87 -0.96 -0.09 2.24 4.31 0.73
q5 -0.44 0.50 2.31 4.39 NA 1.69
q6 -1.43 -0.52 0.34 2.59 5.03 1.2
q7 -1.26 -0.75 0.08 1.77 4.42 0.85

Ordinal to interval transformation table

Ordinal-interval transformation table
Ordinal sum score Logits SE (Logits) Interval score (0-100) SE (0-100) MU (0-100)
1 -3.44 1.01 0.0 9.7 19.4
2 -2.72 0.73 6.9 7.0 14.0
3 -2.27 0.62 11.2 6.0 12.0
4 -1.94 0.55 14.4 5.3 10.6
5 -1.65 0.51 17.2 4.9 9.8
6 -1.41 0.49 19.5 4.7 9.4
7 -1.18 0.47 21.7 4.5 9.0
8 -0.96 0.46 23.8 4.4 8.8
9 -0.76 0.45 25.7 4.3 8.6
10 -0.56 0.45 27.6 4.3 8.6
11 -0.35 0.45 29.6 4.3 8.6
12 -0.15 0.45 31.5 4.3 8.6
13 0.06 0.46 33.6 4.4 8.8
14 0.28 0.47 35.7 4.5 9.0
15 0.51 0.48 37.9 4.6 9.2
16 0.75 0.50 40.2 4.8 9.6
17 1.01 0.52 42.7 5.0 10.0
18 1.28 0.53 45.3 5.1 10.2
19 1.58 0.55 48.1 5.3 10.6
20 1.89 0.57 51.1 5.5 11.0
21 2.22 0.58 54.3 5.6 11.2
22 2.57 0.60 57.6 5.8 11.6
23 2.94 0.61 61.2 5.9 11.8
24 3.32 0.63 64.8 6.1 12.2
25 3.72 0.64 68.6 6.2 12.4
26 4.15 0.66 72.8 6.4 12.8
27 4.60 0.69 77.1 6.7 13.4
28 5.10 0.74 81.9 7.1 14.2
29 5.71 0.84 87.7 8.1 16.2
31 6.99 NA 100.0 NA NA
Note:
SE = Standard Error. Multiply by 2 for MU = Measurement Uncertainty

Item location & infit ZSTD

Item location & outfit ZSTD

First contrast loadings & item locations

The figure above shows the item loadings on the first contrast factor of the PCA of Rasch model residuals, and the item locations.

Software used

Package Version Citation
base 4.1.2 @base
car 3.0.12 @car
cowplot 1.1.1 @cowplot
eRm 1.0.2 @eRm2007b; @eRm2007c; @eRm2009d; @eRm2013e; @eRm2015f; @eRm2019g; @eRm2021a
formattable 0.2.1 @formattable
grateful 0.1.11 @grateful
HH 3.1.47 @HH2004; @HH2014; @HH2015; @HH2022
kableExtra 1.3.4 @kableExtra
knitr 1.39 @knitr2014; @knitr2015; @knitr2022
matrixStats 0.61.0 @matrixStats
mirt 1.36.1 @mirt
mokken 3.0.6 @mokken2007; @mokken2012
psych 2.2.3 @psych
psychotree 0.15.4 @psychotree2010e; @psychotree2011a; @psychotree2015b; @psychotree2018c; @psychotree2018d
reshape 0.8.8 @reshape
RISEkbmRasch 0.1.2 @RISEkbmRasch
rmarkdown 2.14 @rmarkdown2018; @rmarkdown2020; @rmarkdown2022
scales 1.1.1 @scales
tidyverse 1.3.1 @tidyverse

Exercises

  1. Make a table of partipant age distribution
  2. Eliminate either item 1 or 3 and check the effects on reliability
  3. Bonus if you are quick:
    • Run a DIF analysis using age as DIF variable
    • first add the dif.age variable in the setup chunk
    • then duplicate the DIF chunk and modify code

References

Bond, Frank W., Joda Lloyd, and Nigel Guenole. 2013. “Work-Related Acceptance and Action Questionnaire.” American Psychological Association (APA). https://doi.org/10.1037/t72150-000.
Johansson, Magnus. 2021. “Dataset for the Initial Development of the Group Nurturance Inventory.” figshare. https://doi.org/10.6084/M9.FIGSHARE.13042010.V3.
Johansson, Magnus, and Anthony Biglan. 2021. “The Group Nurturance Inventory Initial Psychometric Evaluation Using Rasch and Factor Analysis.” BMC Public Health 21 (1). https://doi.org/10.1186/s12889-021-11474-5.
Strobl, Carolin, Julia Kopf, and Achim Zeileis. 2013. “Rasch Trees: A New Method for Detecting Differential Item Functioning in the Rasch Model.” Psychometrika 80 (2): 289–316. https://doi.org/10.1007/s11336-013-9388-3.